It's recommend that you select the files you want to remove first, otherwise all files will be deleted. You can hold the CTRL or SHIFT key down to select multiple files.
Goto Actions menu and select Delete or click on the Delete speed button in the toolbar.